Skip to content

NUTCH-2887 Migrate to JUnit 5 Jupiter - #862

Merged
lewismc merged 12 commits into
apache:masterfrom
lewismc:NUTCH-2887
Sep 12, 2025
Merged

lewismc merged 12 commits into
apache:masterfrom
lewismc:NUTCH-2887

Conversation

@lewismc

@lewismc lewismc commented Sep 6, 2025 •

Copy link
Copy Markdown
Member

Phase 2 of NUTCH-2887

This PR is scoped only to migrate core tests to JUnit 5. All plugins still run on < JUnit 5. This is intended to make the PR easier to interpret and review. I added the junit-jupiter-engine dependency to ivy.xml and removed the vintage flag from the test-core target in build.xml. The result is that core tests are run as JUnit 5 tests and plugins tests are running as JUnit 4/3 tests.

Comments

  • The following JUnit 3 test cases existed and were first migrated to Unit 4 before being upgraded to JUnit 5.

    • TestParseSegment
    • TestMimeUtil
    • TestAdaptiveFetchSchedule
    • ContinuousCrawlTestUtil
  • In some places Hamcrest utility methods were used to improve the readability of JUnit 5 assertions. This is minor and no additional dependency has been introduced. I used the Hamcrest 1.3.0 transitive dependency in the test scope.

  • I decided to optimize imports in a few classes where it made sense. I think we could consider optimizing imports for the entire Java codebase in a different PR.4

  • Static imports have been used for all JUnit 5 assertions and assumptions

Next steps

The next PR will focus on migrating all plugins tests to JUnit 5.

@lewismc lewismc self-assigned this Sep 6, 2025
@lewismc
lewismc marked this pull request as draft September 6, 2025 01:45
@lewismc

lewismc commented Sep 6, 2025

Copy link
Copy Markdown
Member Author

For some reason the final plugin test TestSlashURLNomrlaizer seems to be having and the last CI job times out after >3 hours! I suppose we should set a timeout configuration for GitHub CI as well as Jenkins. I'll look into that.
I'll investigate the hanging/timeout issue ASAP.

@lewismc

lewismc commented Sep 7, 2025

Copy link
Copy Markdown
Member Author

OK I found an issue. I'm not sure if it is the sole issue but it needs fixed.
Several plugin tests are failing with

```java.lang.NoClassDefFoundError`
...
Caused by: java.lang.ClassNotFoundException
...

Clearly this is a classpath issue. 
JUnit 3 & 4 relied on the `<junit>` ant task which scanned Java source files (`.java`).
JUnit 5 scans for compiled class files. I suspect this is the issue.
I'm working on a solution and will update ASAP.

@sebastian-nagel

Copy link
Copy Markdown
Contributor

It's not TestSlashURLNormalizer but one of the HTTP protocol tests which is hanging:

   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(java.base@11.0.28/Native Method)
        at org.apache.nutch.protocol.AbstractHttpProtocolPluginTest.launchServer(AbstractHttpProtocolPluginT
est.java:208)
        at org.apache.nutch.protocol.AbstractHttpProtocolPluginTest.launchServer(AbstractHttpProtocolPluginT
est.java:227)
        at org.apache.nutch.protocol.httpclient.TestProtocolHttpClient.testNoPreemptiveAuth(TestProtocolHttp
Client.java:112)

Other HTTP protocol tests are failing.

@sebastian-nagel

Copy link
Copy Markdown
Contributor

... and the reason is: AbstractHttpProtocolPluginTest (in core, o.a.n.protocol) is already upgraded to JUnit 5 while the HTTP protocol plugin test classes aren't yet, although they inherit from the abstract test class.

Simplest solution: revert the changes to AbstractHttpProtocolPluginTest until the plugin tests are migrated.

@lewismc

lewismc commented Sep 11, 2025 •

Copy link
Copy Markdown
Member Author

Update on PR

@sebastian-nagel thanks for the comments 👍
I came to the same realization this evening when I was able to pick this issue back up and diagnose further. I just pushed a fairly substantial update which upgrades all plugin unit tests as well as those in core.
I kept the changes in plugin to a minimum, following the same process as for core

  1. identify all JUnit 3 test classes and API usage and upgrade to JUnit 4
  2. Upgrade all JUnit 4 API usage to JUnit 5
  3. Usage static imports for all Assertions

Future work considerations

  1. I haven't made use of @Tag for any unit tests. We could add this later down the line. See 2.10. Tagging and Filtering.
  2. I noticed that unit tests output is now written to a logs directory which may or may not be desirable. Was this always the case? If we want to change this behavior we can adjust the directory that the forked JVM initializes in.
  3. JUnit 4 is still pulled in as a transitive dependency, I believe of MRUnit. I logged https://issues.apache.org/jira/browse/NUTCH-3125 to address this and basically also to get rid of MRUnit if possible.

Let me apologize for the change in plan. I realize this PR it a beast. This was not my intention. I spent too much time working on the hanging Ant process(es) due to complications between JUnit 5 in core and JUnit 3 & 4 in plugin. I decided to not let that get in the way of the bigger picture.

@lewismc
lewismc marked this pull request as ready for review September 11, 2025 07:35

@sebastian-nagel sebastian-nagel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 lgtm.

Tried locally both ant test and ant test-full which also runs two long-running test classes.

One minor point: at least some of the annotations @org.junit.jupiter.api.Test could be written as @Test.

Comment thread src/test/org/apache/nutch/fetcher/TestFetcher.java Outdated
@lewismc
lewismc merged commit e2b60fc into apache:master Sep 12, 2025
4 checks passed
@lewismc
lewismc deleted the NUTCH-2887 branch September 12, 2025 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants